Search for: All records

People use videos to learn new recipes, exercises, and crafts. Such videos remain difficult for blind and low vision (BLV) people to follow as they rely on visual comparison. Our observations of visual rehabilitation therapists (VRTs) guiding BLV people to follow how-to videos revealed that VRTs provide both proactive and responsive support including detailed descriptions, non-visual workarounds, and progress feedback. We propose Vid2Coach, a system that transforms how-to videos into wearable camera-based assistants that provide accessible instructions and mixed-initiative feedback. From the video, Vid2Coach generates accessible instructions by augmenting narrated instructions with demonstration details and completion criteria for each step. It then uses retrieval-augmented-generation to extract relevant non-visual workarounds from BLV-specific resources. Vid2Coach then monitors user progress with a camera embedded in commercial smart glasses to provide context-aware instructions, proactive feedback, and answers to user questions. BLV participants (N=8) using Vid2Coach completed cooking tasks with 58.5\% fewer errors than when using their typical workflow and wanted to use Vid2Coach in their daily lives. Vid2Coach demonstrates an opportunity for AI visual assistance that strengthens rather than replaces non-visual expertise. 

Anticipating how a person will interact with objects in an environment is essential for activity understanding, but existing methods are limited to the 2D space of video frames-capturing physically ungrounded predictions of "what" and ignoring the "where" and "how". We introduce FIction for 4D future interaction prediction from videos. Given an input video of a human activity, the goal is to predict which objects at what 3D locations the person will interact with in the next time period (e.g., cabinet, fridge), and how they will execute that interaction (e.g., poses for bending, reaching, pulling). Our novel model FIction fuses the past video observation of the person's actions and their environment to predict both the "where" and "how" of future interactions. Through comprehensive experiments on a variety of activities and real-world environments in EgoExo4D, we show that our proposed approach outperforms prior autoregressive and (lifted) 2D video models substantially, with more than 30% relative gains. 

Abstract BackgroundLike other oviparous organisms, the gonotrophic cycle of mosquitoes is not complete until they have selected a suitable habitat to oviposit. In addition to the evolutionary constraints associated with selective oviposition behavior, the physiological demands relative to an organism’s oviposition status also influence their nutrient requirement from the environment. Yet, studies that measure transmission potential (vectorial capacity or competence) of mosquito-borne parasites rarely consider whether the rates of parasite replication and development could be influenced by these constraints resulting from whether mosquitoes have completed their gonotrophic cycle. MethodsAnopheles stephensimosquitoes were infected withPlasmodium berghei, the rodent analog of human malaria, and maintained on 1% or 10% dextrose and either provided oviposition sites (‘oviposited’ herein) to complete their gonotrophic cycle or forced to retain eggs (‘non-oviposited’). Transmission potential in the four groups was measured up to 27 days post-infection as the rates of (i) sporozoite appearance in the salivary glands (‘extrinsic incubation period' or EIP), (ii) vector survival and (iii) sporozoite densities. ResultsIn the two groups of oviposited mosquitoes, rates of sporozoite appearance and densities in the salivary glands were clearly dependent on sugar availability, with shorter EIP and higher sporozoite densities in mosquitoes fed 10% dextrose. In contrast, rates of appearance and densities in the salivary glands were independent of sugar concentrations in non-oviposited mosquitoes, although both measures were slightly lower than in oviposited mosquitoes fed 10% dextrose. Vector survival was higher in non-oviposited mosquitoes. ConclusionsCosts to parasite fitness and vector survival were buffered against changes in nutritional availability from the environment in non-oviposited but not oviposited mosquitoes. Taken together, these results suggest vectorial capacity for malaria parasites may be dependent on nutrient availability and oviposition/gonotrophic status and, as such, argue for more careful consideration of this interaction when estimating transmission potential. More broadly, the complex patterns resulting from physiological (nutrition) and evolutionary (egg-retention) trade-offs described here, combined with the ubiquity of selective oviposition behavior, implies the fitness of vector-borne pathogens could be shaped by selection for these traits, with implications for disease transmission and management. For instance, while reducing availability of oviposition sites and environmental sources of nutrition are key components of integrated vector management strategies, their abundance and distribution are under strong selection pressure from the patterns associated with climate change. Graphical Abstract